Faster sorting algorithms discovered using deep reinforcement learning.

Fundamental algorithms such as sorting or hashing are used trillions of times on any given day1. As demand for computation grows, it has become critical for these algorithms to be as performant as possible. Whereas remarkable progress has been achieved in the past2, making further improvements on the efficiency of these routines has proved challenging for both human scientists and computational approaches. Here we show how artificial intelligence can go beyond the current state of the art by discovering hitherto unknown routines. To realize this, we formulated the task of finding a better sorting routine as a single-player game. We then trained a new deep reinforcement learning agent, AlphaDev, to play this game. AlphaDev discovered small sorting algorithms from scratch that outperformed previously known human benchmarks. These algorithms have been integrated into the LLVM standard C++ sort library3. This change to this part of the sort library represents the replacement of a component with an algorithm that has been automatically discovered using reinforcement learning. We also present results in extra domains, showcasing the generality of the approach.

A domain-agnostic approach for characterization of lifelong learning systems.

Despite the advancement of machine learning techniques in recent years, state-of-the-art systems lack robustness to "real world" events, where the input distributions and tasks encountered by the deployed systems will not be limited to the original training context, and systems will instead need to adapt to novel distributions and tasks while deployed. This critical gap may be addressed through the development of "Lifelong Learning" systems that are capable of (1) Continuous Learning, (2) Transfer and Adaptation, and (3) Scalability. Unfortunately, efforts to improve these capabilities are typically treated as distinct areas of research that are assessed independently, without regard to the impact of each separate capability on other aspects of the system. We instead propose a holistic approach, using a suite of metrics and an evaluation framework to assess Lifelong Learning in a principled way that is agnostic to specific domains or system techniques. Through five case studies, we show that this suite of metrics can inform the development of varied and complex Lifelong Learning systems. We highlight how the proposed suite of metrics quantifies performance trade-offs present during Lifelong Learning system development - both the widely discussed Stability-Plasticity dilemma and the newly proposed relationship between Sample Efficient and Robust Learning. Further, we make recommendations for the formulation and use of metrics to guide the continuing development of Lifelong Learning systems and assess their progress in the future.

Latent goal models for dynamic strategic interaction.

Understanding the principles by which agents interact with both complex environments and each other is a key goal of decision neuroscience. However, most previous studies have used experimental paradigms in which choices are discrete (and few), play is static, and optimal solutions are known. Yet in natural environments, interactions between agents typically involve continuous action spaces, ongoing dynamics, and no known optimal solution. Here, we seek to bridge this divide by using a "penalty shot" task in which pairs of monkeys competed against each other in a competitive, real-time video game. We modeled monkeys' strategies as driven by stochastically evolving goals, onscreen positions that served as set points for a control model that produced observed joystick movements. We fit this goal-based dynamical system model using approximate Bayesian inference methods, using neural networks to parameterize players' goals as a dynamic mixture of Gaussian components. Our model is conceptually simple, constructed of interpretable components, and capable of generating synthetic data that capture the complexity of real player dynamics. We further characterized players' strategies using the number of change points on each trial. We found that this complexity varied more across sessions than within sessions, and that more complex strategies benefited offensive players but not defensive players. Together, our experimental paradigm and model offer a powerful combination of tools for the study of realistic social dynamics in the laboratory setting.

Role of perfusion imaging in differentiating multifocal vasospasm-related ischemia versus thromboembolic stroke in a setting of cocaine abuse.

Cerebrovascular complications related to cocaine abuse are reaching epidemic proportions. Contemporary treatments for acute stroke have made it essential to gather all possible diagnostic information before proceeding with intervention. We describe a cocaine abuser who presented with acute right sided neurological deficits and deteriorating mental status. An MRI demonstrated right sided acute and chronic infarcts in the border zones of the right anterior cerebral arteries (ACA) and middle cerebral arteries (MCAs). Subsequent CT angiography (CTA)/CT perfusion (CTP) identified multifocal cerebral vasospasm of the bilateral ACAs and MCAs, preserved cerebral blood volume (CBV) and decreased cerebral blood flow (CBF) in bilateral frontoparietal regions. Early diagnosis of multifocal vasospasm related ischemia directed appropriate therapy and excluded thrombolytic intervention. After 3 weeks, patient's presenting symptoms gradually resolved. We report a unique case of cocaine induced multifocal vasospasm exhibiting late (>3 weeks) reversibility of focal neurological deficits. Furthermore, we illustrate the benefits of CTA/CTP imaging in the setting of cocaine abuse, differentiating multifocal vasospasm induced hypoperfusion/ischemia from focal thromboembolic ischemia/infarct and allowing for appropriate medical management in the crucial hyperacute setting.